0x14. JavaScript - Web scraping
Scripting - API - JavaScript
Intro.
JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write and easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language, Standard ECMA-3rd Edition - December 1999. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal data-interchange language.
How to Manipulate JSON Data:
1. Reading JSON from a File: To read data from a JSON file, we first load the content of the file using the 'fs' module, which stands for "file system." Then, we convert this content into a usable JavaScript object. It's like opening a book and reading its contents.
Here's the code snippet with explanations:
'utf8': This argument specifies the encoding of the file. 'utf8' is the most common encoding for text files, and it ensures that the data is read as a text string.Writing to files:
The
fs.writeFile()function in Node.js is a convenient way to write content to a file. It simplifies the process of opening, writing, and closing a file:Using
fs.writeFile():const data = 'Hello, world!'; fs.writeFile('path/to/file.txt', data, (err) => { if (err) throw err; // Content written successfully });GET request in Node.js
To make a GET request in Node.js using the `
request` module, you need to install the `request` module first.Here's an example of how to make a GET request using the `request` module:
1. Install the `request` module: Open your terminal or command prompt and run the following command to install the `request` module:
```bash
npm install request```
2. Require the `request` module: In your Node.js file, require the `request` module at the top.
```javascript
const request = require('request');```
3. Make the GET request: Use the `request()` function to make a GET request. This function takes an options object as a parameter, where you specify the URL you want to request.
```javascript
const url = 'http://example.com';request(url, (error, response, body) => {// Handle the response});```
In the above example, `'http://example.com'` is the URL you want to request. You can replace it with the actual URL you want to make a GET request to.
4. Handle the response: In the callback function, you can handle the response from the server. The `response` object represents the response and provides properties like `statusCode` and `headers`. The `body` parameter contains the response body.
```javascript
request(url, (error, response, body) => {if (error) {console.error('Error making GET request:', error);return;}// Handle the response
console.log('Status code:', response.statusCode);console.log('Response body:', body);});
```
In the above example, if an error occurs during the request, it will be logged to the console. Otherwise, you can access the response properties like `statusCode` and the response body.
Sending a GET request with a specific ID
JSON.parse() to convert it into a JavaScript object. For example:Resulting Object: After calling
JSON.parse(), theparsedObjectvariable will contain a JavaScript object with the same key-value pairs as the original JSON string. In this case,parsedObjectwould be:
{ name: 'John', age: 30 }Use Cases: JSON parsing is commonly used in scenarios where you receive data in JSON format, such as when making HTTP requests to APIs. The JSON response from the server is initially a string, and you use
JSON.parse()to convert it into a JavaScript object so that you can work with the data as a regular object in your code.
Here are some common use cases for JSON.parse():
When working with APIs: You often receive JSON responses from APIs, and you use
JSON.parse()to extract data from these responses.When reading JSON from files: If you have JSON data stored in a file, you can read it as a string and then parse it using
JSON.parse()to work with the data in your application.
Notes:
err,res, andbodyare commonly used callback parameters returned by therequestmodule or other HTTP request libraries. Here's what each of these parameters represents:
err(Error): This parameter typically holds an error object if an error occurred during the HTTP request. If there were no errors, it's usuallynull. You should checkerrto handle errors gracefully. For example, iferris notnull, it might indicate a network issue, an invalid URL, or a server-side error.res(Response):resrepresents the HTTP response object. It provides information about the response, such as the status code, headers, and more. You can inspect properties likeres.statusCodeto check the HTTP status code. It is usually an instance of thehttp.IncomingMessageclass in Node.js.body(Response Body):bodycontains the actual content of the HTTP response, typically as a string. It is often in the form of the response data, such as JSON or HTML content. You can usebodyto access and work with the response data after a successful request..
Notes in practice:
Access command-line arguments in Node scripts:
In a Node.js script, you can access command-line arguments using the process.argv array. The process.argv array contains command-line arguments passed to your script. Here's how to access and use these arguments:
Accessing Command-Line Arguments:
process.argv[0]contains the path to the Node.js executable.’For example, on a Unix-like system, it might look like
/usr/local/bin/node”process.argv[1]contains the path to the script being executed.“For example, if you run your script with
node myScript.js,process.argv[1]will contain the path tomyScript.js”So, In practice, when working with command-line arguments, you usually start accessing and processing arguments from
process.argv[2], as this is where the actual command-line arguments provided after the script name begin.
Nice to remember :
Accessing command-line arguments varies between programming languages and shell scripting. Here's a quick comparison of how to access command-line arguments in Python, Node.js, and a Bash script:
1. **Python:**
In Python, command-line arguments are available in the `sys.argv` list from the `sys` module.
```python
import sys
# Accessing command-line arguments in Python
script_name = sys.argv[0]
arguments = sys.argv[1:]
print(f"Script Name: {script_name}")
print(f"Arguments: {arguments}")
```
2. **Node.js:**
In Node.js, you can access command-line arguments via the `process.argv` array.
```javascript
// Accessing command-line arguments in Node.js
const scriptName = process.argv[1];
```
3. **Bash:**
In a Bash script, you can access command-line arguments directly as positional parameters using variables like `$1`, `$2`, and so on.
```bash
# Accessing command-line arguments in a Bash script
script_name=$0
argument1=$1
argument2=$2
echo "Script Name: $script_name"
echo "Argument 1: $argument1"
echo "Argument 2: $argument2"
```
In all three cases, the actual command to access command-line arguments is quite similar: you reference variables or elements in an array to access the arguments provided to the script. The specifics may vary slightly, such as indexing, but the concept is the same.
Comments in Node:
In Node.js, you can use two types of comments: single-line comments and multi-line comments. Here's how to use both types of comments in Node.js:
// This is a single-line comment in Node.js
======================
/*
This is a
multi-line comment
in Node.js
*/How to write the response body of an HTTP request into a file in Node.js:
To write the response body of an HTTP request into a file in Node.js, you can use the fs (File System) module. Here's an example of how to do this:
We make an HTTP request using the
requestmodule to the specified URL.If the request is successful (status code 200), we use the
fs.writeFilemethod to write thebody(the response body) into a file namedresponse.txt. You can change the filename and path to whatever you prefer....Note:
Encoding: The
formatvariable should be set to'utf8'for the file encoding.fs.writeFileParameters: The parameters forfs.writeFileshould be in order of :fs.writeFile(path, data, options, callback).
Create a writable stream for writing data to a file:
In Node.js, `createWriteStream` is a method provided by the built-in `fs` (File System) module. It is used to create a writable stream for writing data to a file.
This method is particularly useful when you want to write a large amount of data to a file efficiently or when you want to stream data from one source to a file.
Here's how you can use `createWriteStream` and an explanation of its key components:
```javascript
const fs = require('fs');
// Create a writable stream to write data to a file
const writableStream = fs.createWriteStream('output.txt');
// Write data to the stream
writableStream.write('Hello, World!\n');
writableStream.write('This is a new line.');
// Close the stream when done
writableStream.end();
```
Explanation of `createWriteStream` and its key components:
1. **Import `fs` Module**: First, you need to import the built-in `fs` module in Node.js to access file system-related functions.
2. **Create a Writable Stream**: You use `createWriteStream` to create a writable stream. In the example, it creates a writable stream to a file named 'output.txt'. This stream will be used to write data to that file. You can specify the path to the file as an argument to `createWriteStream`.
3. **Write Data to the Stream**: You can use the `write` method of the writable stream to write data to it. In the example, two lines of text are written to the stream. You can write strings, buffers, or other data types.
4. **End the Stream**: After you have written all the data you need, you can call the `end` method on the writable stream to signal that you have finished writing. This method is essential to properly close the stream.
`createWriteStream` is especially useful when working with large files or when streaming data from another source, like reading data from a network or another file and writing it to a destination file in smaller chunks. It allows for efficient, non-blocking, and scalable file writing in Node.js applications.
More in : https://www.geeksforgeeks.org/node-js-fs-createwritestream-method/
Resources: