They have a point. The distro is only carrying a small fraction of what's available on PyPI, and will have out-of-date versions that may lack security patches (and for projects like yt-dlp, are highly unlikely to work at all). Realistically they aren't applying security backports; there would rarely be a reason not to just use the new version, and the resources just aren't there for that scale of patching anyway.
Further, pip will interfere with your system package manager's tracking of the system environment (PEP 668), and any attempt you make to install from .deb/.rpm/etc. to a virtual environment is a hack at best. Maybe you could symlink individual package folders to the system environment while installing a superset, but ugh. Oh, and if you're a developer (like most "Python die-hards" are) then this isn't very helpful, as you aren't testing the installation of your package the way that your users will do it.
So your trusted base is the Python runtime and maybe a few choice libraries that actually have engaged package management. Sadly, it is a trap to think you can put things on the internet as a small player by leveraging lots of third-party stuff, when you don't have the resources to actually maintain that large dependency base.
This also may mean you want a base OS like Fedora and not some crusty "enterprise" variant that tries to bundle a frozen Python from 15 years ago. This is getting a little better in recent years, with some enterprise platforms having optional package streams for parallel install of newer runtime versions, etc.
Edit to add: on the Fedora-derived platforms, pip doesn't really interfere much anymore. The platform python-* RPMs go into /usr/lib/... while pip installs into /usr/local/lib/... or your ~/.local/lib/... hierarchy. The only mysterious behavior you get is if you manage to install in both places and then you have to understand the path search order. Just like if you do strange things with user and virtual environments.
Basically once you leave the realm of the 5-10 most popular building blocks in an area (which OS packagers do distribute), you're still left with plenty of packages that have 10ks+ of users, sane maintainer/security/ownership models, and code that would be very difficult to just copy-paste a la stackoverflow (e.g. native/compiled deps with complex wheel build tooling).