Problem with a tutorial in freeCodeCamp

Hi,

regarding this tutorial: Web Scraping in Python – How to Scrape an eCommerce Website Using Beautiful Soup and Pandas

i use pycharm and pandas and soup are installed (i checked)
i try follow the tut but the parsing on the productlist doesnt pass the javascript I think

If i try to print just the k variable, i get some html, scripts tags but I think I should see with this print the table and so on // and I do not

what is the issue please ?

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
    

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <h1 class="zone-name-title h1">
            <img class="heading-favicon" src="/favicon.ico"
                 onerror="this.onerror=null;this.parentNode.removeChild(this)">
            www.thewhiskyexchange.com
        </h1>
        <h2 class="h2" id="challenge-running">
            Checking if the site connection is secure
        </h2>
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/captcha/nojs/transparent.gif?ray=788787cadf7e6283')"></div>
        <div id="challenge-body-text" class="core-msg spacer">
            www.thewhiskyexchange.com needs to review the security of your connection before proceeding.
        </div>
        <form id="challenge-form" action="/c/35/japanese-whisky?__cf_chl_f_tk=C6OoMk32fxteH.WTzhGDFRJIL76gMmSUSjMY3KFdQ90-1673543686-0-gaNycGzNB2U" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="acSvikW9.EPt7jLmTQ7sqUcEVS3_rszXNfL4.80VHE0-1673543686-0-ARKQUVrhbPtdymeygUHQ7yfSL7u9vB1noQHie7lVYxOdcCuzVLfGFSd4EVs0Yph2z7z-W6GoA3D_zBdYKkFDMhibR2Sg4iiJUrMl1WLkMGOqmA8HpkMPLQljKUpHDwORWsGFMbm14etQ86lJ99EZwWEzR3PnPoGMWED_qAL9Kt9bjtF9OV0kd4vTUj8xLZZW5ISqrD-7mbJJu_mHE5ZAgDxsLJ9fksUaNDaj0hIsPoEG3dV9D-ejVoH3n5KaU34szNpv3gzYvt25Ok8VpRmn5eWZjNaw2_Rz1KnlpAPDgbLfLVpNTsu0n44hJ7kn7WU1MakqfPgFHAIam3llGTU5GS09BOW2F0CU5eJDabKCgQEM73AMOHifn-dcaVgiEmXeTtd6Aatc_C3APnZyILlKBT51GD6en9OxPS_sX-T1xYhWqN85dB2O2FQir5gZxGJtgphQiA5p_v-qq8TlwGJaCNW6tp_pwB72JGZRAMv5J97L_81Q9DiD3epaLMDyrkLFqfAdnJHVpHNbmhrm9tTTlxcH13Revq4n9uV8uBRg-V1Yb3-pDGWbymcHULz1EpN_IA">
            <input type="hidden" name="r" value="EBy78VypT1IwYEB50U9iEvGo6FChl0eCDryHK3lno5k-1673543686-0-Abjq+JTjhTrr1TLIuCgbinR2KtzL3kmXAjFEy6qKFj9344bM+l/IQj0FYHYG43BwAnbcKrl+TWHeuWwyXsGh/Sqj31ag5U+TcQOnNXoMr6h9lnE7coJO5+nkhOcvuBBLziycHzn9NfBhP2DHtDmNdT5AZpD86XKaXq40pAAMs0bDZmaLKj9+lNAy5ews/Xt3dtRM9B+uvg59l+a/KrwnEalZz85pInqckNFsJYOIyLBH/qaxGO/cVwvfVk1Tl6EzZ79eHcVfh5iDM8MqKWIwaQE8fuqrBmWhI5KyZuVywfy8ZQlzTvA/mbZ1Lxl8nWT12XctfmO1OP1HccxDmjlSAp6KSFPFFLZr6xKKJidpLpvu5Rij17G1atKbzSEIabe6Pkj7ceTVJZNr3w1c/vdkZK1fptMq2EG41CziaOtxYA5h4Ngsvuu3R+ohAu12QFTWFNJNwvn1/ObENMswFLTLgKWc/pY1/a1H6dk9r+olfbZu25GJLkSptUT5QVGHtlpuPALLi8fdKhqiJQ28KbHbFtt7MJyuD11oJaCecC17e/JoXg6kmvNbUvmHcBZW1u3sNF81Bf9U0Hh3bSE+yqZDwKL/QXMW7XHYayHCzfFtFgnoF3ahpa4iDXnp1tNW9lJys5G0BSFnCqbNeBxIGjY07WU4+1C/hHLV9ouX1ivGeFl5HwGU1I3InvvHvpjgGGqGRC9AmvrplgbkGK6b/M2TT+J7TgSW34mSwVSpInX07RZwE337PPpNUxDvwxaYaClfN/d2SZ6+TxXdEZIB1aiHk0+/oFNn6Znkyfly/MZ5ApDZsbWqdHbXAl3Ipz95Sm+Oyv6aJCRFVvuxifAs/pMdcoX0zft5mJ3n8EtPoxOZFQz5kZZ5DX6r/UDouu8t7YOMMBak2F6x2bByYVdsu9Y1gSnt7Z/tShiN9mUvKf2l16PhUWWBK/cwSsV9j2gpTKef4aQj1i1q5+QCDeA0U5K0ESH/YpU09erFfm8KmlfVy9eTwyqDLfF4bpEwl9j4pz0zvC7nE+Ie61lsqg6nkjmeEJnxfUtsquQmnivzxvqlcHRAwJb768P2edGUUxj3f9Id+TeKJJ3XQZK6I1ASPIDcromZmiCBFdOzkSsgL8CrxkJWm4e7zg5GqTAF1KCvogZz4oF9z1DCHJ/VuOQzrzndAlJdFpcmXry+BKBZbS67teJaukqppXRk2Q+L2VxfdwywRmM8ecWpcUBpTEHUJcBJ3aUwn+JLLVeWFL+isBHmCG6fLI+roEoml49a/k+ZYK1zZOoegoq09acBnZ3qmOrB12zIbyzwG59smiJYUPpFdAFvkMb+RrDHyGSivtsu60oq74FM3VNA92g6tmSvweuiRh7K5RvmqZFx9aCkcGqNBItDgYf3aFQtDLufxBFl/DpGRcdQZun49wV4PtmXqpISnGczfnng5cVkB4/AAiPB8E3OLHmlxnlJIazdgOpGxZDtr1cuO1bidY/nwhtKt/sB6viXb3TdEJ0a5HMZa9QIRuOXdJ2Dxk4a7gjAS48XCsr8Q2mV/2JHsrPdOxBPfOIy21Omx9/N6P/QizmXk15UWpyyZIeNKv+tmQM23L+T/2lnoFvpI0jCFZOpW49D8p0Ubz8aa2s3GKQ6s/EYc7u5eOZkyBe/RPVcPYOlrJY9C4rDclNoeFoYWZdYWwJld7oIySLLPemzwDGihyY00n3Z/u4ejN/YAPnjCQxNiqp8B0MppKxhZBpMJjI28UObrtKyOGnT3wuolg/3tujE7LL09iXk0GHFejFwCpJ+84uBF2wGvHsqwT0Dhqwng09w7j9/Fe6EJVgQD39ycDy05Xm9XnrC">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cType: 'interactive',
            cNounce: '62499',
            cRay: '788787cadf7e6283',
            cHash: '163d59ef6e68e06',
            cUPMDTk: "\/c\/35\/japanese-whisky?__cf_chl_tk=C6OoMk32fxteH.WTzhGDFRJIL76gMmSUSjMY3KFdQ90-1673543686-0-gaNycGzNB2U",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cTplV: 4,
            cTplB: 'cf',
            cRq: {
                ru: 'aHR0cHM6Ly93d3cudGhld2hpc2t5ZXhjaGFuZ2UuY29tL2MvMzUvamFwYW5lc2Utd2hpc2t5',
                ra: 'cHl0aG9uLXJlcXVlc3RzLzIuMjguMg==',
                rm: 'R0VU',
                d: 'WAdgwGvrbE8RXZGvKInAQcTitTIe02wWJpNy9k7bEvEVJW7KhhJORGxdNRmn/kB5UqoXBmER/xecxhzO4NJGO+qcBT6Vh8xvhlWYDiw9DcYsiiYDCQhSfZuocWa9PL1UY3c1bIFz/2P/pNCRGHGSkahnlz9NvGacu0NpGyYzFSbOCgomozC/cBq+e0NQPsAm23dsCx+Gy1RE+o+KyZIyKRAndo+46UGbRVNzPOpx2vw00zSbnpFse5Mr3rltuFczCafR78GygMRnBMilEgcFrJnK5rdF/Ky0egAWsjDkrvaNHQgtcYJkDpTmwUgO+hqnH4EX7N4t+sfCE529qBHkCaVmx8BZ0biNlG6oJgjN67KWNBUtY4eE34AbKgyhgeCLmBN4sjWXNQhqpbGt8HzVxquYldw8HCOWWyVWdERIr8FFW9zg/7EnFYNhIKna6KnPttp4tgg8WmcSqC0ADQPqwsu3EfTsMjUzgTYW8ZC+bVF1lSCViX7ktlfHkl7+GglRX/oSuna3Qfk9+ESO1f1E+M47e2ZWTLYb820EGu66vHBwqMh/oesrpFEEca3K9iAdpnLf8wJOSINyrkm8YG/S6Jb0qSN3EC9o5b00VNjg3HmIpBhaaRGCl6ubjrpsW+sateazdGNG9PZ9/2KVwH2+//eF+cQDg9SuGJgcZkqEaiI=',
                t: 'MTY3MzU0MzY4Ni44ODEwMDA=',
                m: 'HBkm1dnQbx6qDjDU/D98eHF9yVIZBLfO4Tn+aCAMH/c=',
                i1: 'c3GiY+K/wWFwWUGqWdAM+g==',
                i2: '8sSGqJ/GuLG9QeIG4FQzAQ==',
                zh: 'Lu11UPKpct80h4UpHjnFr7549vZ5EGi2V7KjmFdfcoc=',
                uh: 'neMFmDCz0LRc0/Wea+5x0IhJieHbzcQVSkMgFkgnrjc=',
                hh: '1iL0YmRIkuteIeg9zu2NzR9zkRexldgCzJYoCkFcSM8=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/captcha/js/transparent.gif?ray=788787cadf7e6283');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1?ray=788787cadf7e6283';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/c\/35\/japanese-whisky?__cf_chl_rt_tk=C6OoMk32fxteH.WTzhGDFRJIL76gMmSUSjMY3KFdQ90-1673543686-0-gaNycGzNB2U" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


    <div class="footer" role="contentinfo">
        <div class="footer-inner">
            <div class="clearfix diagnostic-wrapper">
                <div class="ray-id">Ray ID: <code>788787cadf7e6283</code></div>
            </div>
            <div class="text-center">Performance &amp; security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=l" target="_blank">Cloudflare</a></div>
        </div>
    </div>
</body>
</html>


Process finished with exit code 0

I solved in the meantime. I made a default headers var and did this and it worked:

k = requests.get('https://www.thewhiskyexchange.com/c/35/japanese-whisky', headers=DEFAULT_HEADERS).text

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.