isOkay to not use dns.lookup for microservice short_url challenge?

I’ve been trying for days to try to figure out how to use dns.lookup since it is suggested to use it in the challenge. I’m sure I put in the correct arguments, but I’m not for sure how to use it to check if the url is vaild.
I replace https:// and http:// before putting it as the first argument to dns.lookup(hostname, options, callback);

The reason I took off https… and http… was because it was triggering an error.

The hostname parameter seems to expect either www.websitename.com or websitename.com
but when I manually input other values into the hotname parameter such as “23423423” or etc, dns.lookup returns an object with a property called hostname with the value “23423423” …
…is it suppose to do that? or am I doing something wrong?

Since the 4th test in the challenge is " If you pass an invalid URL that doesn’t follow the valid http://www.example.com format, the JSON response will contain { error: 'invalid url' }",
And since I couldn’t get dns.lookup to work, I decided to search the user inputted string for http and https, and if it didn’t include it, the code would yield {error: “invalid url”} and return, exit the function.

If you would like to look at my code, it’s below, but since I was testing and trying to figure out things I have a lot of commented out code (I’m Sorry if it makes it harder for you to look through it in anyway. Initially I try to figure things out, test things, from memory or experiment to see if I can get it to work before trying to look online or elsewhere for information).

mongoose.connect(process.env.MONGO_URI, {useNewUrlParser: true, useUnifiedTopology: true}); // FCC added the second property: value pair

db = mongoose.connection;

db.on("error", console.error.bind(console, "There Has Been A Connection Error!"));

db.on("open", function() {
  console.log(db.readyState === 1, "The Database Is Connected. TYG.")

  // The rest of the code:

  const urlSchema = new mongoose.Schema({
    url: String,
    ID: Number
  });

  const Url = mongoose.model("Url", urlSchema);

  app.route("/api/shorturl/new")
  .post(async function shortUrl(req, res) {

    if (!req.body.url.includes("http")) {
      res.json({error: "invalid url"});
      return;
    }

    const urlParameter = await req.body.url;

    const hostname = await req.body.url.replace(/https?:\/\//, "");

    // res.json({newURL: new URL(req.body.url)}) returns "https://www.noelbray.com/"

    let validUrl = await dns.lookup(hostname, {hints: dns.ADDRCONFIG}, function(err, address, family) {
      if (err) console.log(err)
      // if (err) return err;
      console.log("address: ", address, address === null)
      // return address;
      // if(address) {
      //   res.json({test: address});
      // }
      // resolve(address);
      // return res.json({dnstest: address === null})
      // if (address === null || address === undefined) res.json({error: "invalid"});
      // validUrl = address;
      // return res.json({address: address || "I'm trying to figure this out."});
      // res.send("ok");
    });

    //  if (validUrl) { 
    //   // return res.json(validUrl.hostname);
    //   // return res.json({error: "invalid url"})
    //   return res.json({test: validUrl});
    // } else {
    //   return res.json({error: "invalid url"})
    //   // return res.json(validUrl.code)
    // }

    // if(/https:\/\/wwww|http:\/\/www/.test(validUrl.hostname)) {
    //   return res.json(validUrl.family);
    // } else {return {error: "invalid url"}};
    
    // res.json(url);

    const urlExist = await Url.exists({url: urlParameter});

    // res.json("I'm working. " + urlParameter + " " + urlExist);  
    // TTG, I figured out the line above was causing this error: "UnhandledPromiseRejectionWarning: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client" 
    // Explaination Which basically is telling you that you cannot send more than one res.json to a specific route, path 
    // article: https://stackoverflow.com/questions/52122272/err-http-headers-sent-cannot-set-headers-after-they-are-sent-to-the-client

    if(urlExist) {
    
      const doc = await Url.findOne({url: urlParameter});

      res.json({
        original_url: doc.url,
        short_url: doc.ID
      })
    
    } else {

      const shortUrlNumber = Math.floor(Math.random() * 10001);

      const idExist = await Url.exists({ID: shortUrlNumber});

      if (idExist) shortUrlNumber = Math.floor(Math.random() * 10001);
    
      const url = new Url({
        url: urlParameter,
        ID: shortUrlNumber
      });

      url.save(function (err, url) {
        if(err) return console.log(err);
      });

      // const doc = Url.findOne({url: url});

      res.json({
        original_url: urlParameter,
        short_url: shortUrlNumber
      }); 

    }

    // console.log(dns.lookup(req.body.url, (err, hostname) => {
    //   // if(err) return console.log(err);
    //   console.log("hostname is: ", hostname);
    // }).hostname);

  // left of with trying to figure out how to create an entry in the database. 

  }); // End of post method, request

  app.get("/api/shorturl/:ID", async function (req, res) {
    const short_url = +req.params.ID;

    // I was getting an error Url not defined because I had placed this get request out of the db.on("open", function .... ) event handler function

    // res.json(short_url) // the endpoint route, path is working

    const short_urlExist = await Url.exists({ID: short_url});

    if (short_urlExist) {
      const { url } = await Url.findOne({ID: short_url});

      // res.json(url); // test to see if it was working
      res.redirect(url);
    } 
    else {
      res.json(`The short url ${short_url} does not exists, please try again.`)
    }
  }); // End of get method request

})// End of mongoose.connect db.on("open", function ... )event handler function

The error object returned is just showing you the hostname argument that was passed in.

Wouldn’t you just do the res.json inside the if (err) block as it won’t be executed unless an error was thrown which is what it does if it can’t resolve the address (technically it can throw for other reasons).

Base on your suggestion, I tried:

await dns.lookup(hostname, {hints: dns.ADDRCONFIG}, function(err, address, family) {
      // if (err) console.log(err)
      if(err) {
      res.json({error: "invalid url"});
      return;
      }
    });

And I ran the test, and it ended up passing the 4th test. But failed the 2 middle test.

I’m guessing it’s because of the http check you have.

As far as I can see the URL should be allowed to include http(s) because it is being tested against a URL that includes https.

Thank you for trying to help me.
I commented out the http check, but it still fails the 2 middle test.
I put the https? checker in there so that people would have to type in https or http… based on the placeholder hint value and because the 4th test says that a valid url should have http in it.

Your right I didn’t read your code correctly (missed the ! on req.body.url.includes(“http”)) and was thinking you were excluding http/s, not checking for it.

You have a bunch of commented out code, what are you using and not using? Can you post the code as you have it when you’re submitting it?

@lasjorg

I’ve been trying some other things to get it to work with dns.lookup so here’s the new code, but I’m running into problems with it too:

db.on("error", console.error.bind(console, "There Has Been A Connection Error!"));

db.on("open", function() {
  console.log(db.readyState === 1, "The Database Is Connected. TYG.")

  const urlSchema = new mongoose.Schema({
    url: String,
    ID: Number
  });

  const Url = mongoose.model("Url", urlSchema);

  app.route("/api/shorturl/new")
  .post(async function shortUrl(req, res) {

    // I will probably have uncomment the 4 lines of code just below because it seems that res.redirect() needs http://www. or https://www. to be able to redirect to the url
    // if (!req.body.url.includes("http")) {
    //   res.json({error: "invalid url"});
    //   return;
    // } // this works to pass the 4th test

    const urlParameter = await req.body.url;

    const hostname = await req.body.url.replace(/https?:\/\//, "");

   let invalidUrl = "";
    
    await dns.lookup(hostname, function(err, address, family) {
   
    if (err) {
        console.log("dns.lookup error: ", err)
        invalidUrl = "true";
        res.json({error: "invalid url"});
        return;
      };

    // I added dns.reverse() based on an example I found that is suppose show how to validate a url
    // here's the link to it: https://www.tutorialspoint.com/nodejs/nodejs_dns_module.htm

      dns.reverse(address, function (err, hostnames) {
      if (err) {
        invalidUrl = "true";
        console.log("dns.reverse error:", err);
        res.json({error: "invalid url"});
        return;
      }
    });  

      console.log("invalidUrl", invalidUrl);
    });

     if (invalidUrl === "true") {
      return;
    }

    const urlExist = await Url.exists({url: urlParameter});

    if(urlExist) {
    
      const doc = await Url.findOne({url: urlParameter});

      res.json({
        url: "retrieved from database",
        original_url: doc.url,
        short_url: doc.ID
      })
    
    } else {

      const shortUrlNumber = Math.floor(Math.random() * 10001);

      const idExist = await Url.exists({ID: shortUrlNumber});

      if (idExist) shortUrlNumber = Math.floor(Math.random() * 10001);
    
      const url = new Url({
        url: urlParameter,
        ID: shortUrlNumber
      });

      url.save(function (err, url) {
        if(err) return console.log(err);
      });

       res.json({
        newUrl: "created in database",
        original_url: urlParameter,
        short_url: shortUrlNumber
      }); 

    }

  }); // End of post method, request

  app.get("/api/shorturl/:ID", async function (req, res) {
    const short_url = +req.params.ID;

 
    const short_urlExist = await Url.exists({ID: short_url});

    if (short_urlExist) {
      const { url } = await Url.findOne({ID: short_url});

      res.redirect(url);
    } 
    else {
      res.json(`The short url ${short_url} does not exists, please try again.`)
    }
  }); // End of get method request

})// End of mongoose.connect db.on("open", function ... )event handler function

I assume you have some require statements, mongoose.connect, and are mounting the body parsing middleware at the top?

It might be better if you share the repl.it link instead. As long as your DB connection string is in a .env file only the owner of the repl can get to it.

Hello there,

Just to add: This challenge has recently been altered. Currently, the use of dns.lookup should not be used to achieve the functionality of the 4th user story.

dns.lookup is a hint to add as an optional extra, and works by (essentially) pinging a domain, and either returning a response or a bad gateway.

The 4th user story only expects the http|https format to be checked.

I hope this clarifies.

As for if you should use dns.lookup or not, I have no idea. But you have to strip off everything after the domain name before checking it.

// doesn't work
const hostname = "timestamp-microservice.freecodecamp.rocks/api/timestamp/1605313713457"

// works
const hostname = "timestamp-microservice.freecodecamp.rocks"

As far as I can tell, it seems to work just fine for checking the test URL ftp:/john-doe.org so I’m not sure why you can’t use it?

Thank You Sky020. That’s good to know, because I was able to pass the 4th test without using dns.lookup by checking the user-inputed string with:

 if (!req.body.url.includes("http")) {
       res.json({error: "invalid url"});
       return;
 } // this works for me to pass the 4th test 

I temporarily commented out that code in order to see if I could get all 4 test to pass by using dns.lookup, but I have been unsuccessful in this.

For future reference, when I tried dns.lookup(hostname, cb), and saved the returned value into a variable, it was an object with error properties and a hostname property; and the address parameter within the callback was returning null and undefined for some values.

Also, when I typed in just a number into the input, dns.lookup seem to still allowed it, within my code to be inputed into my database.

@lasjorg & @Sky020 here’s my repl.it link for it:

This is what I ended up using which uses dns.lookup and passes all:

Hint
app.route('/api/shorturl/new').post((req,res)=> {
  ...
    let urlMatch = // Match the URL to expected structure or not

    // Lookup the matched URL
    dns.lookup(urlMatch, (err,add,fam) => {
      // If the URL does not exist, return expected error
      if (err) return res.json({"error": "invalid URL"});
   
      // Save to database, otherwise.
    })
  });
});

It looks as though this is where your issue is:

const hostname = await req.body.url.replace(/https?:\/\//, "");

It is what @lasjorg mentioned here:

Hope that helps

@Sky020 & @lasjorg Thank you for your help. I greatly appreciate it. :slight_smile: