URL Shortener Microservice - Retrieving URL from Database and Shortening It?

[DragonOsman]

What’s happening:

I need to know how to send the GET request to retrieve the URL from the database after the POST request to send it there. Also, should I do app.get in the callback for app.post?

And how do I get the shortened URL?

My code so far

require("dotenv").config();
const express = require("express");
const cors = require("cors");
const path = require("path");
const bodyParser = require("body-parser");
const app = express();
const mongoose = require("mongoose");
const dns = require("dns");

mongoose.connect(process.env.DB_URI, { useNewUrlParser: true, useUnifiedTopology: true });

const Schema = mongoose.Schema;
const urlSchema = new Schema({
  urlCode: String,
  longUrl: String,
  shortUrl: String
});

const Url = mongoose.model("url", urlSchema);

// Basic Configuration
const port = process.env.PORT || 3000;

app.use(cors());

app.use("/public", express.static(`${process.cwd()}/public`));

app.get("/", (req, res) => {
  res.sendFile(path.join(`${process.cwd()}`, "/views/index.html"));
});

// Your first API endpoint
app.get("/api/hello", (req, res) => {
  res.json({ greeting: "hello API" });
});

app.listen(port, () => {
  console.log(`Listening on port ${port}`);
});

// to parse POST request body
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

let originalUrl;
app.post("/api/shorturl/new", (req, res) => {
  originalUrl = req.body.url;
  const url = new Url({
    original_url: originalUrl
  });

  dns.lookup(originalUrl, (err) => {
    if (!err) {
      const saveUrlInDatabase = (done) => {
        url.save((err, url) => {
          if (err) {
            return done(err);
          }
          console.log(`url ${url} saved to database`);
          return done(null, url);
        });
      };

      saveUrlInDatabase((err, data) => {
        if (err) {
          console.log(err);
        }
        if (!data) {
          console.log("Missing done() argument");
        }
      });
    }

    app.get((req, res) => {
      const retrieveUrl = (url, done) => {
        Url.find({ original_url: url }, (err, url) => {
          if (err) {
            done(err);
          }
          done(null, url);
        });
      };

      retrieveUrl(originalUrl, (err, data) => {
        if (err) {
          console.log(err);
        }
        if (!data) {
          console.log("Missing done() argument");
        }
      });
    });
  });
});

Am I passing done correctly? And when do I use res.json to send the JSON response?

My browser information:

User Agent is: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36 Edg/88.0.705.81.

Challenge: URL Shortener Microservice

Link to the challenge:

freeCodeCamp.org

[DragonOsman]

You didn’t say if calling app.get inside app.post's callback is okay and if I’m passing done correctly. Please tell me that stuff as well.

Also, am I using dns.lookup correctly?

Thanks for the info you did give me, though.

[Sky020]

Hello there,

@DragonOsman

Also, am I using dns.lookup correctly?

Not quite. dns.lookup accepts a domain as its first argument. However, you are passing the full URL to it.

@DragonOsman

should I do app.get in the callback for app.post

Again, not quite. Remember, all you are doing when you write app.get/app.post is defining a route. It does not make sense to define one inside the other. In fact, I would imagine this would cause some very unexpected behaviour.

Hope this clarifies

[DragonOsman]

How would the app.get callback know about the URL and Mongoose document I want to use, then?

What do I do to shorten the URL?

[Sky020]

@DragonOsman

How would the app.get callback know about the URL and Mongoose document I want to use, then?

This is what the get request looks like: /api/shorturl/<short_url>
So, the URL wanted is in the short_url parameter.

@DragonOsman

What do I do to shorten the URL?

This can be done in many ways. To keep mine simple, I just used the number of documents in my database. That is, every document in the db had a number related to the its position in the db.

I think you might be making this much more complicated than need be. I suggest you sit back from your keyboard for a few minutes just paying attention to the user stories, and simplifying the workflow.

Keep it up, and I am sure it will click

[DragonOsman]

So I need a URL parameter? The POST request is to /api/shorturl/new and the GET request is to /api/shorturl/<short_url>? Or are they both supposed to be to the latter?

I’m thinking I’ll just use the first 2 or so digits of the _id. Or would that be a bad idea?

[Sky020]

@DragonOsman

So I need a URL parameter?

You do not need a URL parameter. You have a URL parameter. If you are confused by the syntax, then I think this lesson will help:

• Basic Node and Express - Get Route Parameter Input from the Client | Learn | freeCodeCamp.org

@DragonOsman

I’m thinking I’ll just use the first 2 or so digits of the _id . Or would that be a bad idea?

This would be a bad idea, because there could be duplicates.

[DragonOsman]

@Sky020

This would be a bad idea, because there could be duplicates.

Then I should probably assign a number to each document based on order of creation. I was already thinking of that from the beginning but I don’t know how to do it.

For the GET request, I just have to get the original URL from the database and make a redirect to it, right?

Then given this code:

app.post("/api/shorturl/new", (req, res) => {
  const originalUrl = req.body.url;
  const domain = new URL(originalUrl).hostname;
  const url = new Url({
    original_url: originalUrl,
    short_url: 
  });

  dns.lookup(domain, (err) => {
    if (err) {
      res.json({ error: err });
    }

    const saveUrlInDatabase = (urlDoc, done) => {
      urlDoc.save((err, url) => {
        if (err) {
          return done(err);
        }
        console.log(`url ${url} saved to database`);
        return done(null, url);
      });
    };

    saveUrlInDatabase(url, (err, data) => {
      if (err) {
        console.log(err);
      }
      if (!data) {
        console.log("Missing done() argument");
      }
    });
  });
});

app.get("/api/shorturl/:short_url", (req, res) => {
  const retrieveOriginalUrl = (url, done) => {
    Url.find({ original_url }, (err, url) => {
      if (err) {
        done(err);
      }
      done(null, url);
    });
  };

  retrieveOriginalUrl(req.params.short_url, (err, data) => {
    if (err) {
      console.log(err);
    }
    if (!data) {
      console.log("Missing done() argument");
    }
  });
});

how do I get the original URL from here on the GET request? I’m not clear on that. And aside from the short_url key’s value, which I don’t have yet, how is my POST code so far?

Also, where in the POST request part should I send the response with the original and short URLs in it? At the bottom of the dns.lookup callback?

[Sky020]

@DragonOsman

For the GET request, I just have to get the original URL from the database and make a redirect to it, right?

Correct.

@DragonOsman

if (err) {
      res.json({ error: err });
    }

Remember, the user story is:

If you pass an invalid URL that doesn’t follow the valid http://www.example.com format, the JSON response will contain { error: 'invalid url' }

How do you know what err will hold?

---

@DragonOsman

how do I get the original URL from here on the GET request?

You have the original URL in your database…you need to find it, when someone says “take me to short_url 4”, then you find what “short_url 4” represents in your database, and redirect.

@DragonOsman

where in the POST request part should I send the response with the original and short URLs in it?

Anywhere… Provided there are no errors in the request (and your logic), and you have the data to send, you can just res.json it. Logically, you have steps to follow:

1. Accept URL in POST request
2. Ensure URL is of the correct format (what dns.lookup is doing)
3. Save URL to database with associated shortened version
4. Respond with what you saved it as.

Remember, try not to overcomplicate things

Hope this helps

[DragonOsman]

How do I get the position of a document in the database? I’m sorry if this is a noob question.

[Sky020]

@DragonOsman

How do I get the position of a document in the database?

To be honest. I am not sure. My understanding is that you cannot for Non-relational databases. Could very-well be wrong.

You should not need this, though.

[DragonOsman]

I’m just trying to ask you how you did that:

@Sky020

To keep mine simple, I just used the number of documents in my database. That is, every document in the db had a number related to the its position in the db.

I thought I should try this too but how do I do it? Do I make an array and use the array indices?

[Sky020]

What I did was something like this (not actual code, but you get the just):

const short_url = Model.find({}).length; // number of documents in db
const newDoc = new Model({short_url, original_url}); // create new doc

newDoc.save(cb); // Save to db

Then, to get the original URL:

const givenShortURL = req.params.short_url; // From GET req
Model.findOne({short_url: givenShortURL}, cb); // Find and redirect in cb

[DragonOsman]

What am I still doing wrong here?

// to parse POST request body
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

app.post("/api/shorturl/new", (req, res) => {
  const originalUrl = req.body.url;
  const domain = new URL(originalUrl).hostname;
  const shortUrl = Url.find({ original_url: originalUrl }).length;
  const url = new Url({
    original_url: originalUrl,
    short_url: shortUrl
  });

  dns.lookup(domain, (err) => {
    if (err) {
      res.json({ error: "invalid url" });
    }

    const saveUrlInDatabase = (urlDoc, done) => {
      urlDoc.save((err, url) => {
        if (err) {
          return done(err);
        }
        console.log(`url ${url} saved to database`);
        return done(null, url);
      });
    };

    saveUrlInDatabase(url, (err, data) => {
      if (err) {
        console.log(err);
      }
      if (!data) {
        console.log("Missing done() argument");
      }
    });

    res.json({
      original_url: originalUrl,
      short_url: shortUrl
    });
  });
});

app.get("/api/shorturl/:short_url", (req, res) => {
  let doc;
  const retrieveOriginalUrl = (url, done) => {
    doc = Url.findOne({ short_url: req.params.short_url }, (err, url) => {
      if (err) {
        done(err);
      }
      done(null, url);
    });
  };

  retrieveOriginalUrl(req.params.short_url, (err, data) => {
    if (err) {
      console.log(err);
    }
    if (!data) {
      console.log("Missing done() argument");
    }
  });
  console.log(doc);
});

I still don’t seem to be getting the URL into the database. What’s logged by me doing console.log(url ${url} saved to database); is url { _id: 60454ad3bcdcdb05cc267529, __v: 0 } saved to database which shows neither the short URL nor the long one.

Any help is appreciated. Thanks.

Edit: The response I get in my browser for the GET request doesn’t have the short URL in it either. Just the original one.

[Sky020]

@DragonOsman

const shortUrl = Url.find({ original_url: originalUrl }).length;

There are a few issues here.

@Sky020

(not actual code, but you get the just):

1. Model.find does not return an array
2. Model.find is asynchronous. (same goes for all db operations)

You need to be working with promises, when you do these operations.

---

What is the point of this:

@DragonOsman

const retrieveOriginalUrl = (url, done) => {

@DragonOsman

retrieveOriginalUrl(req.params.short_url, (err, data) => {

@DragonOsman

Url.findOne({ short_url: req.params.short_url }, (err, url) => {

You are getting there, but really take the time to work through this. Take a break, if you need one. Otherwise, you could end up introducing more bugs than you resolve.

[DragonOsman]

retrieveOriginalUrl should get the long URL from the database. Did I do it wrong?

For the short URL, I tried this:

let shortUrl;
  Url.find({ original_url: originalUrl })
    .then(data => console.log(data))
    .then(res => {
      console.log(res);
    })
    .catch(err => console.log(err));

I get undefined for one of those console.log calls.

[Sky020]

@DragonOsman

Url.find({ original_url: originalUrl })
    .then(data => console.log(data))
    .then(res => {
      console.log(res);
    })

Let us walk through this.

1. Url.find returns a promise of some data
2. Once the promise resolves, the first .then gets it in its callback
3. Now, once data => console.log(data) resolves, the return is sent to the next .then
4. console.log() always returns undefined
5. res => console.log(res) - res is defined as the return of the previous .then which happens to be undefined

I suggest you go through this section of the curriculum, for a refresher on promises, and arrow functions:

• Learn ES6: Write Arrow Functions with Parameters | freeCodeCamp.org
• Learn ES6: Create a JavaScript Promise | freeCodeCamp.org

Hope this helps

[DragonOsman]

I tried this as well for the code I showed earlier:

Url.find({ original_url: originalUrl })
    .then(data => data)
    .then(res => {
      for (const elem of res) {
        console.log(elem);
      }
      return res;
    })
    .catch(err => console.log(err));

This still doesn’t work.

I also need to know why the response returned from the POST request didn’t have a short url even when I didn’t have that commented out here:

const url = new Url({
    original_url: originalUrl//,
    // short_url: shortUrl
  });

[Sky020]

@DragonOsman

.then(data => data)

This is doing nothing. Practically, nothing anyway; it is the same as this function:

function myFunction(someParameter) {
  return someParameter;
}

Also, why are you finding based on the original_url? You do not know the original URL. You do know the short URL.

[DragonOsman]

@Sky020 I’m really confused. Why does the database not get any URL fields despite me adding them in my schema? What am I doing wrong there?

Here’s my current code:

app.post("/api/shorturl/new", (req, res) => {
  const originalUrl = req.body.url;
  const domain = new URL(originalUrl).hostname;
  let shortUrl;
  Url.find({}, "original_url", (err, urls) => {
    if (err) {
      console.log(err);
    }
    console.log(urls);
    shortUrl = urls.length;
  });
  const url = new Url({
    original_url: originalUrl,
    short_url: shortUrl
  });

  dns.lookup(domain, (err) => {
    if (err) {
      res.json({ error: "invalid url" });
    }

    const saveUrlInDatabase = (urlDoc, done) => {
      urlDoc.save((err, url) => {
        if (err) {
          return done(err);
        }
        console.log(`url ${url} saved to database`);
        return done(null, url);
      });
    };

    saveUrlInDatabase(url, (err, data) => {
      if (err) {
        console.log(err);
      }
      if (!data) {
        console.log("Missing done() argument");
      }
    });

    res.json({
      original_url: originalUrl,
      short_url: shortUrl
    });
  });
});

app.get("/api/shorturl/:short_url", (req, res) => {
  let doc;
  const retrieveOriginalUrl = (url, done) => {
    doc = Url.findOne({ short_url: req.params.short_url }, (err, url) => {
      if (err) {
        done(err);
      }
      done(null, url);
    });
  };

  retrieveOriginalUrl(req.params.short_url, (err, data) => {
    if (err) {
      console.log(err);
    }
    if (!data) {
      console.log("Missing done() argument");
    }
  });
  console.log(doc);
});

Did I do the done stuff incorrectly?