I'm 2 to 3 hours into The Witcher 3 and drowners are impossible to kill. What am I doing wrong?

In addition to what @PausePause said, you should make sure the enemies you're fighting aren't too high-level for you.

You can easily check by looking at the color of the number to the left of the enemy's health bar. According to the game manual:

8. Enemy Health Bar: Displays the currently targeted enemy’s name, level, and remaining health. The health bar’s color indicates the targeted enemy’s type: silver indicates an enemy of magic origin, red indicates all other enemy types. The number to the left of the bar shows the enemy’s level. If you are 5 or more levels above the enemy, the number will be gray. If you are within 4 levels of the enemy, the number will be green. If the enemy is 5 or more levels above you, the number will be red. A skull near the health bar indicates the enemy is a major threat.

Emphasis mine.

Here are a few examples of what this looks like:

Green

Red

Skull

Enemies far above your own level receive very little damage, unless you're striking their weakness. Drowners are supposed to be among the weakest of monsters, so you may have wandered into an area that's beyond your current level. This means that every enemy — not just Drowners — will be difficult to defeat.

You have 2 swords: the steel one and the silver one. The silver sword is for killing monsters. Geralt should pull the correct sword out automatically. Check if you manually pulled out the wrong weapon.

The start of The Witcher 3 is pretty rough. Basically you need to follow the main story line until you have gained some levels; the level requirement of the next main line quest may be too high for you. Before this point, every monster is tough for Geralt.