The above squares the result of plus1(y), so if y is equal to 3, then plus1(3) returns 4. The square of 4 is 16. If you are getting 13, please post all of the code.
What you wrote on the right side of the = is not what is happening with square(plus1(y)). The value of x does not come into play here. The plus1(3) is evaluated first and passed to the square function, so it is really just sqare(4) which returns 16.