Why do Pillagers keep spawning near my singleplayer Minecraft house? Is there anything I can do about it?

From the wiki:

Patrols can spawn during the day on blocks with a block light level of 7 or lower. If a player is within two chunks of a village, patrols do not spawn.

The game picks a random location 24–48 blocks away from the player to try to spawn the patrol's captain.

Additionally, in bedrock there are additional location where Illager (patrols) can spawn, where other mobs cannot - this includes the river biome, but I am not certain how this works in Java.

You can check around the area where they normally spawn and see if there are any cave entrances nearby. Using F3 you can see the "light level" that a block has.

That's natural for Pillagers. They're meant to spawn randomly so you can get village bonuses and start raids. Light level means nothing in this scenario as they are an overworld mob. All they do naturally is come out and attack at random points, just like Wandering Traders spawn randomly. The best you could do is fight them off, but there is, sadly, no solution besides redstone traps.

EDIT:

Pillagers continually spawn in a 72×54×72 block volume centered on the top floor. They may spawn on any valid opaque block as long as the sky light level is 11 or lower, and the block light level is 8 or lower, but can also spawn on grass or sand blocks regardless of block light level.

Love, the official Minecraft Wiki. This disproves the light level theorum.

They spawn on all grass and sand, regardless of light level. Blocking it up with torches is not only a waste of materials, but ineffective as well. They will spawn in any of the 5184 blocks of dirt around your base. Place a crap-ton of netherrack or something around your base, but I doubt you could block the 5184 blocks you would have to place around your base you would need to prevent them from spawning. Just deal with it at this point. I would take pillager raids as awesome because they give you free stuff.