Can't figure out why dot product isn't facing the direction I think it will

I’m trying to make it so when the player hits attack, they swing (and check for targets) in the direction they’re holding the WSAD keys/analog stick. So if the player hits attack while holding D, the game will check for hits in the cone starting in the model’s transform.right:

for (int i = 0; i < defenderList.Count; i++){ //Iterate through the list of enemies that are in range

			if (Input.GetAxis ("Horizontal") == 0 && Input.GetAxis ("Vertical") == 0){		//If the player isn't holding any movement key
				playerAttackAngle = Vector3.Dot(directionToTarget, transform.forward);		//Take the dot product of the direction to target +  player's forward
			}
			else{																			//If the player IS holding a movement key
				playerAttackAngle = (Vector3.Dot(directionToTarget,  new Vector3(Input.GetAxis ("Horizontal"), 0, Input.GetAxis ("Vertical")) ) );		//Take the dot product of the direction to target plus the direction of the vector formed by the WSAD keys the player is holding 
			}
			if (distanceToTarget <= attack.range && (playerAttackAngle > attack.areaOfEffect)){				//Double check that enemy is in range, and inside the player's attack cone
				//Message enemy that they've been hit
			}
		}

However, I can’t seem to make this work: enemies are correctly hit if I stand in front of them and don’t move, but when I hold any direction, enemies in the direction I’m holding don’t register the hit. Am I doing my vector math wrong?

Well, your problem is probably that you did not normalize both vectors, did you? The dot product is literally just:

public static float Dot(Vector3 V1, Vector3 V2)
{
    return V1.x * V2.x  +  V1.y * V2.y  +  V1.z * V2.z;
}

So to use the result as cos(alpha) the vectors need to be normalized. GetAxis might return a value smaller than 1 (since it’s slowly increasing, depending on the axis settings in the Input manager) also if you press both directions at the same time, both axis will be 1, so the length of your vector will be sqrt(2) which will also give you a wrong Dot-product result.

Simply use “.normalize” after each vector when performing the dot product:

playerAttackAngle = Vector3.Dot(directionToTarget.normalized,  (new Vector3(Input.GetAxis ("Horizontal"), 0, Input.GetAxis ("Vertical"))).normalized );

Finally, since we don’t know what value is stored in “attack.areaOfEffect”, the reference value need to be something smaller than 1 since cos(0°) == 1 and cos(90°) == 0. If you really have an actual angle in “attack.areaOfEffect” you have to use Mathf.Acos to get the angle. In this case keep in mind that Mathf.Acos returns the angle in radians not in degree. To convert it to degree you can simply multiply the result by Mathf.Rad2Deg