Line of Sight with Physics.Linedraw or Physics.Raycast Unreliable?

Alright, so I'm trying to implement a basic LoS algorithm here and while it seems pretty obvious at first, something is just not right - Physics.Linedraw (or Raycast) seem quite flakey in returning consistent LoS detection.

I have a screenshot showing a case where I'm clearly within LoS but its not being picked up (green lines = LoS, red lines = no LoS).

http://grab.by/47J0

If I walk around it constantly flips between having LoS or not. If your behind something obvious, such as the hill, at least that seems consistent - so there must be some kind of artifact causing this. When I use a 3DText over the heads of red line npc's, it always comes back as the hit.collider.name == "Terrain", yet its clear that terrain is not directly causing this.

Heres the simplified code below, I'm aware its costly and inefficient, right now my focus is in trying to get this working 100% consistently ... it's probably something silly but I'm at a loss:

    // TODO: This MUST be moved into a place thats only called when PC's MOVE or Environment Changes!  Calling in Update() will be insanely expensive
private void CheckLoS()
{
    // walk over each non-player controlled entity in scene (TODO: needs culling with range check)
    foreach(GameObject subject in GameObject.FindGameObjectsWithTag("NPC"))
    {
        // check against each player controlled entity in scene
        foreach (GameObject player in GameObject.FindGameObjectsWithTag("Player"))
        {
            // add +2.0f to y-axis to account for height of player/npc (hardcoded for now, top of rigidbody/collider later)
            Vector3 playerPosition = player.transform.position + new Vector3(0.0f, 2.0f, 0.0f);
            Vector3 subjectPosition = subject.transform.position + new Vector3(0.0f, 2.0f, 0.0f);
            if (!Physics.Linecast(playerPosition, subjectPosition))
            {
                Debug.DrawLine(playerPosition, subjectPosition, Color.green);
            }
            else
            {
                Debug.DrawLine(playerPosition, subjectPosition, Color.red);
            }
        }
    }
}

One thing I notice that your script may not be accounting for is that the linecast will collide with the NPC's collider also because Unity is checking if a line between one position and another collides with anything. So even if there is a LoS to your NPC from your player, it will return false because the player's colliders will be hit. So you have to change you linecast to:

RaycastHit hit;

if(Physics.Linecast(playerPosition, subjectPosition, hit)) {
     if(subject.collider == hit.collider) 
          Debug.DrawLine(playerPosition, subjectPosition, Color.green);

     else  
          Debug.DrawLine(playerPosition, subjectPosition, Color.red);
}

else {
    Debug.DrawLine(playerPosition, subjectPosition, Color.green);//not sure what this would be if we missed, because we should always hit something.
}