Future, it is best to come to a decision how you need the crawler to take care of different Website systems. You can find an ongoing debate with regards to the intelligence of search engine crawlers. It isn't fully apparent if they are full-blown headless browsers or just glorified https://youtu.be/EVeTP0aV8gY?si=5FHbTy4kbLwUVoJD