Hi,
regarding this tutorial: Web Scraping in Python – How to Scrape an eCommerce Website Using Beautiful Soup and Pandas
i use pycharm and pandas and soup are installed (i checked)
i try follow the tut but the parsing on the productlist doesnt pass the javascript I think
If i try to print just the k variable, i get some html, scripts tags but I think I should see with this print the table and so on // and I do not
what is the issue please ?
<!DOCTYPE html>
<html lang="en-US">
<head>
<title>Just a moment...</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<meta name="robots" content="noindex,nofollow">
<meta name="viewport" content="width=device-width,initial-scale=1">
<link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
</head>
<body class="no-js">
<div class="main-wrapper" role="main">
<div class="main-content">
<h1 class="zone-name-title h1">
<img class="heading-favicon" src="/favicon.ico"
onerror="this.onerror=null;this.parentNode.removeChild(this)">
www.thewhiskyexchange.com
</h1>
<h2 class="h2" id="challenge-running">
Checking if the site connection is secure
</h2>
<noscript>
<div id="challenge-error-title">
<div class="h2">
<span class="icon-wrapper">
<div class="heading-icon warning-icon"></div>
</span>
<span id="challenge-error-text">
Enable JavaScript and cookies to continue
</span>
</div>
</div>
</noscript>
<div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/captcha/nojs/transparent.gif?ray=788787cadf7e6283')"></div>
<div id="challenge-body-text" class="core-msg spacer">
www.thewhiskyexchange.com needs to review the security of your connection before proceeding.
</div>
<form id="challenge-form" action="/c/35/japanese-whisky?__cf_chl_f_tk=C6OoMk32fxteH.WTzhGDFRJIL76gMmSUSjMY3KFdQ90-1673543686-0-gaNycGzNB2U" method="POST" enctype="application/x-www-form-urlencoded">
<input type="hidden" name="md" value="acSvikW9.EPt7jLmTQ7sqUcEVS3_rszXNfL4.80VHE0-1673543686-0-ARKQUVrhbPtdymeygUHQ7yfSL7u9vB1noQHie7lVYxOdcCuzVLfGFSd4EVs0Yph2z7z-W6GoA3D_zBdYKkFDMhibR2Sg4iiJUrMl1WLkMGOqmA8HpkMPLQljKUpHDwORWsGFMbm14etQ86lJ99EZwWEzR3PnPoGMWED_qAL9Kt9bjtF9OV0kd4vTUj8xLZZW5ISqrD-7mbJJu_mHE5ZAgDxsLJ9fksUaNDaj0hIsPoEG3dV9D-ejVoH3n5KaU34szNpv3gzYvt25Ok8VpRmn5eWZjNaw2_Rz1KnlpAPDgbLfLVpNTsu0n44hJ7kn7WU1MakqfPgFHAIam3llGTU5GS09BOW2F0CU5eJDabKCgQEM73AMOHifn-dcaVgiEmXeTtd6Aatc_C3APnZyILlKBT51GD6en9OxPS_sX-T1xYhWqN85dB2O2FQir5gZxGJtgphQiA5p_v-qq8TlwGJaCNW6tp_pwB72JGZRAMv5J97L_81Q9DiD3epaLMDyrkLFqfAdnJHVpHNbmhrm9tTTlxcH13Revq4n9uV8uBRg-V1Yb3-pDGWbymcHULz1EpN_IA">
<input type="hidden" name="r" value="EBy78VypT1IwYEB50U9iEvGo6FChl0eCDryHK3lno5k-1673543686-0-Abjq+JTjhTrr1TLIuCgbinR2KtzL3kmXAjFEy6qKFj9344bM+l/IQj0FYHYG43BwAnbcKrl+TWHeuWwyXsGh/Sqj31ag5U+TcQOnNXoMr6h9lnE7coJO5+nkhOcvuBBLziycHzn9NfBhP2DHtDmNdT5AZpD86XKaXq40pAAMs0bDZmaLKj9+lNAy5ews/Xt3dtRM9B+uvg59l+a/KrwnEalZz85pInqckNFsJYOIyLBH/qaxGO/cVwvfVk1Tl6EzZ79eHcVfh5iDM8MqKWIwaQE8fuqrBmWhI5KyZuVywfy8ZQlzTvA/mbZ1Lxl8nWT12XctfmO1OP1HccxDmjlSAp6KSFPFFLZr6xKKJidpLpvu5Rij17G1atKbzSEIabe6Pkj7ceTVJZNr3w1c/vdkZK1fptMq2EG41CziaOtxYA5h4Ngsvuu3R+ohAu12QFTWFNJNwvn1/ObENMswFLTLgKWc/pY1/a1H6dk9r+olfbZu25GJLkSptUT5QVGHtlpuPALLi8fdKhqiJQ28KbHbFtt7MJyuD11oJaCecC17e/JoXg6kmvNbUvmHcBZW1u3sNF81Bf9U0Hh3bSE+yqZDwKL/QXMW7XHYayHCzfFtFgnoF3ahpa4iDXnp1tNW9lJys5G0BSFnCqbNeBxIGjY07WU4+1C/hHLV9ouX1ivGeFl5HwGU1I3InvvHvpjgGGqGRC9AmvrplgbkGK6b/M2TT+J7TgSW34mSwVSpInX07RZwE337PPpNUxDvwxaYaClfN/d2SZ6+TxXdEZIB1aiHk0+/oFNn6Znkyfly/MZ5ApDZsbWqdHbXAl3Ipz95Sm+Oyv6aJCRFVvuxifAs/pMdcoX0zft5mJ3n8EtPoxOZFQz5kZZ5DX6r/UDouu8t7YOMMBak2F6x2bByYVdsu9Y1gSnt7Z/tShiN9mUvKf2l16PhUWWBK/cwSsV9j2gpTKef4aQj1i1q5+QCDeA0U5K0ESH/YpU09erFfm8KmlfVy9eTwyqDLfF4bpEwl9j4pz0zvC7nE+Ie61lsqg6nkjmeEJnxfUtsquQmnivzxvqlcHRAwJb768P2edGUUxj3f9Id+TeKJJ3XQZK6I1ASPIDcromZmiCBFdOzkSsgL8CrxkJWm4e7zg5GqTAF1KCvogZz4oF9z1DCHJ/VuOQzrzndAlJdFpcmXry+BKBZbS67teJaukqppXRk2Q+L2VxfdwywRmM8ecWpcUBpTEHUJcBJ3aUwn+JLLVeWFL+isBHmCG6fLI+roEoml49a/k+ZYK1zZOoegoq09acBnZ3qmOrB12zIbyzwG59smiJYUPpFdAFvkMb+RrDHyGSivtsu60oq74FM3VNA92g6tmSvweuiRh7K5RvmqZFx9aCkcGqNBItDgYf3aFQtDLufxBFl/DpGRcdQZun49wV4PtmXqpISnGczfnng5cVkB4/AAiPB8E3OLHmlxnlJIazdgOpGxZDtr1cuO1bidY/nwhtKt/sB6viXb3TdEJ0a5HMZa9QIRuOXdJ2Dxk4a7gjAS48XCsr8Q2mV/2JHsrPdOxBPfOIy21Omx9/N6P/QizmXk15UWpyyZIeNKv+tmQM23L+T/2lnoFvpI0jCFZOpW49D8p0Ubz8aa2s3GKQ6s/EYc7u5eOZkyBe/RPVcPYOlrJY9C4rDclNoeFoYWZdYWwJld7oIySLLPemzwDGihyY00n3Z/u4ejN/YAPnjCQxNiqp8B0MppKxhZBpMJjI28UObrtKyOGnT3wuolg/3tujE7LL09iXk0GHFejFwCpJ+84uBF2wGvHsqwT0Dhqwng09w7j9/Fe6EJVgQD39ycDy05Xm9XnrC">
</form>
</div>
</div>
<script>
(function(){
window._cf_chl_opt={
cvId: '2',
cType: 'interactive',
cNounce: '62499',
cRay: '788787cadf7e6283',
cHash: '163d59ef6e68e06',
cUPMDTk: "\/c\/35\/japanese-whisky?__cf_chl_tk=C6OoMk32fxteH.WTzhGDFRJIL76gMmSUSjMY3KFdQ90-1673543686-0-gaNycGzNB2U",
cFPWv: 'g',
cTTimeMs: '1000',
cTplV: 4,
cTplB: 'cf',
cRq: {
ru: 'aHR0cHM6Ly93d3cudGhld2hpc2t5ZXhjaGFuZ2UuY29tL2MvMzUvamFwYW5lc2Utd2hpc2t5',
ra: 'cHl0aG9uLXJlcXVlc3RzLzIuMjguMg==',
rm: 'R0VU',
d: 'WAdgwGvrbE8RXZGvKInAQcTitTIe02wWJpNy9k7bEvEVJW7KhhJORGxdNRmn/kB5UqoXBmER/xecxhzO4NJGO+qcBT6Vh8xvhlWYDiw9DcYsiiYDCQhSfZuocWa9PL1UY3c1bIFz/2P/pNCRGHGSkahnlz9NvGacu0NpGyYzFSbOCgomozC/cBq+e0NQPsAm23dsCx+Gy1RE+o+KyZIyKRAndo+46UGbRVNzPOpx2vw00zSbnpFse5Mr3rltuFczCafR78GygMRnBMilEgcFrJnK5rdF/Ky0egAWsjDkrvaNHQgtcYJkDpTmwUgO+hqnH4EX7N4t+sfCE529qBHkCaVmx8BZ0biNlG6oJgjN67KWNBUtY4eE34AbKgyhgeCLmBN4sjWXNQhqpbGt8HzVxquYldw8HCOWWyVWdERIr8FFW9zg/7EnFYNhIKna6KnPttp4tgg8WmcSqC0ADQPqwsu3EfTsMjUzgTYW8ZC+bVF1lSCViX7ktlfHkl7+GglRX/oSuna3Qfk9+ESO1f1E+M47e2ZWTLYb820EGu66vHBwqMh/oesrpFEEca3K9iAdpnLf8wJOSINyrkm8YG/S6Jb0qSN3EC9o5b00VNjg3HmIpBhaaRGCl6ubjrpsW+sateazdGNG9PZ9/2KVwH2+//eF+cQDg9SuGJgcZkqEaiI=',
t: 'MTY3MzU0MzY4Ni44ODEwMDA=',
m: 'HBkm1dnQbx6qDjDU/D98eHF9yVIZBLfO4Tn+aCAMH/c=',
i1: 'c3GiY+K/wWFwWUGqWdAM+g==',
i2: '8sSGqJ/GuLG9QeIG4FQzAQ==',
zh: 'Lu11UPKpct80h4UpHjnFr7549vZ5EGi2V7KjmFdfcoc=',
uh: 'neMFmDCz0LRc0/Wea+5x0IhJieHbzcQVSkMgFkgnrjc=',
hh: '1iL0YmRIkuteIeg9zu2NzR9zkRexldgCzJYoCkFcSM8=',
}
};
var trkjs = document.createElement('img');
trkjs.setAttribute('src', '/cdn-cgi/images/trace/captcha/js/transparent.gif?ray=788787cadf7e6283');
trkjs.setAttribute('style', 'display: none');
document.body.appendChild(trkjs);
var cpo = document.createElement('script');
cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1?ray=788787cadf7e6283';
window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
if (window.history && window.history.replaceState) {
var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
history.replaceState(null, null, "\/c\/35\/japanese-whisky?__cf_chl_rt_tk=C6OoMk32fxteH.WTzhGDFRJIL76gMmSUSjMY3KFdQ90-1673543686-0-gaNycGzNB2U" + window._cf_chl_opt.cOgUHash);
cpo.onload = function() {
history.replaceState(null, null, ogU);
};
}
document.getElementsByTagName('head')[0].appendChild(cpo);
}());
</script>
<div class="footer" role="contentinfo">
<div class="footer-inner">
<div class="clearfix diagnostic-wrapper">
<div class="ray-id">Ray ID: <code>788787cadf7e6283</code></div>
</div>
<div class="text-center">Performance & security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=l" target="_blank">Cloudflare</a></div>
</div>
</div>
</body>
</html>
Process finished with exit code 0