International Seminar of Research Month 2021
Volume 2022
http://dx.doi.org/10.11594/nstp.2022.2426
How to cite:
Diyasa et al. (2022). Push notification using the websocket in the application of Sistem Informasi Uji Kompetensi Online
(Situk) Version 2. International Seminar of Research Month 2021. NST Proceedings. pages 154-163. doi: 10.11594/
nstp.2022.2426
Conference Paper
Push Notification Using the Websocket in the Application of Sistem Informasi
Uji Kompetensi Online (Situk) Version 2
I Gede Susrama Mas Diyasa
1
*, Gideon Setya Budiwitjaksono
2
, Alfiatun Masrifah
1
, Muhammad
Rif'an Dzulqornain
1
1
Department of Informatics, Faculty of Computer Science, Universitas Pembanguna Nasional “Veteran” Jawa
Timur, Surabaya 60294, Indonesia
2
Departments of Accounting, Faculty of Economics and Business, Universitas Pembanguna Nasional
“Veteran” Jawa Timur, Surabaya 60294, Indonesia
*Corresponding author:
ABSTRACT
SITUK-V2 is an application of the second version of the online competency
test integrated system, which is used in the competency test at the LSP
(Lembaga Sertifikasi Profesi) UPN "Veteran" Jawa Timur, with the signing up
or registration process up to a statement of competence. At the time of
registration, both from the session of the asesi and the asesor and the
statement of graduation by the asesor, the asesi will receive a notification in
the registered email, this notification uses push notification technology. Push
notification technology is a technology that allows a device (client) to get
information in the form of notifications sent by the information provider
(server) automatically. In general, the mechanism for sending notifications
from the server to the client uses a stand-alone push service (broker) and
uses the Hypertext Transfer Protocol (HTTP) protocol. but in sending
notifications there is often a delay to the client. In this study, a notification
delivery mechanism with push notifications was developed which will be
implemented using the WebSocket protocol in the SITUK-V2 application
where the result is that there is a difference in delay time between the
registration process and the competency statement process that is sent via
user email, this is due to the influence of the size of the data, bandwidth, and
device used.
Keywords: Push Notifications, WebSocket, SITUK
E-mail:
[email protected]
Introduction
Lembaga Sertifikasi Profesi (LSP) University of Pembangunan Nasional “Veteran” Jawa Timur
is an institution that conducts competency assessments and competency certification for students
according to professional skills that have been licensed by the Badan Nasional Sertifikasi Profesi
(BNSP). Competency Certification is the process of providing competency certificates that are
carried out systematically and objectively through competency exams that refer to the
certification scheme that has been made by LSP and approved by BNSP. The processes that occur
in LSP are quite numerous and require good data processing. The processes that occur at LSP are
carried out manually, which causes these activities to run slowly and the recorded data is not
automatically connected to other parts, and data related to the assessment is not stored neatly.
With these problems, it is necessary to change the LSP data processing method from manual to
computerized and automatic LSP data processing method. Where it can carry out the participant
registration process, independent assessment by asesi, data maintenance by admin or LSP
employees, and the process of determining graduation become systemized and integrated.
ISRM 2021
155
The competency test process begins with registration, both collectively and independently by
visiting the LSP of the University of Pembangunan Nasional “Veteran” Jawa Timur. This
registration will later provide information that will be submitted to the asesor (the asesor is a
team of examiners in the competency exam) or admin, which will then be notified to the asesi (the
asesi is a competency test participant) if approved to take the competency test, where the
implementation of the professional certification competency test consists of several stages,
namely written, verbal, practice, interview, simulation, and others. If after the competency test,
and the asesi is declared competent, the asesi will receive a certification that is recognized by the
Badan Nasional Sertifikasi Profesi (BNSP).
When an asesi or asesor (all of them are users) performs the registration process, either by
the asesor or by the asesi and a statement of competence by the asesor, the asesi and asesor will
receive a notification feature to be able to receive information quickly and periodically without
having to open the site or application first. This notification feature is sent from the server using
push notification technology. Push notification is a technology for sending information in the form
of notifications from an information provider (server) to a device (client) automatically (Hansen
et al., 2012). The notification received by the client is in the form of a successful registration
notification and a statement of competence. The working system of this technology sends
notifications by directly sending notifications from the server to the client without having to be
asked by the client first (Brüstel & Preuss, 2012). This notification delivery mechanism is arranged
in a service called a push service. By using this technology, device users do not need to visit sites
or applications periodically to get information or notification results. If there is the latest
information from the site or application, the device user will immediately get a notification that
there is the latest information from the site or application. Examples of this push technology are
SMS notifications, Email notifications, and others. By using this technology, device users do not
need to visit sites or applications periodically to get information or notification results. If there is
the latest information from the site or application, the device user will immediately get a
notification that there is the latest information from the site or application. Examples of this push
technology are SMS notifications, Email notifications, and others. By using this technology, device
users do not need to visit sites or applications periodically to get information or notification
results. If there is the latest information from the site or application, the device user will
immediately get a notification that there is the latest information from the site or application.
Examples of this push technology are SMS notifications, Email notifications, and others.
Several studies related to push notifications, among others, were conducted by Elliot Estep
(Estep, 2013). One aspect that Elliot Estep wants to achieve is to know the performance of the
protocol being tested on the client-side. The parameter used is resource usage on the client-side.
Likewise, research conducted by Muhammad et al. 2018) compared the Performance of the
WebSocket Protocol with the SSE Protocol on Push Notification Technology, where both protocols
will be implemented and tested based on 3 scenarios, namely 1 client, 3 clients, and 6 clients. The
results obtained are the average delay and CPU usage on the SSE protocol is smaller than the
WebSocket protocol. Yudianto et al. (2017) developed a notification delivery mechanism with
push notifications that will be implemented using the WebSocket protocol and broker
implementation into the server. The results of the development in this study have a maximum
number of clients that can connect to the server as many as 1015 clients and the average delay
obtained from testing the sending of notifications in bulk, sending notifications of different sizes,
and sending notifications using different bandwidths.
Based on the case study above, the research that will be carried out is to see or find out the
performance of the WebSocket protocol in the application of Sistem Terintegrasi Uji Kompetensi
Online (SITUK) with push notification technology. The research will be conducted using the flutter
framework with Dart language and javascript, the Socket.IO library, on the scope of the internet
network via the https://situk-web.igsindonesia.org/ server and for clients using laptops or PCs
and smartphones with operating systems Android and iOS. The test will be carried out using the
ISRM 2021
156
delay parameter, the results obtained will be analyzed to determine the performance of the
WebSocket protocol related to the delivery method based on the test parameters.
Literature Review
Push Notifications
Push notification is included in the category of internet communication model, based on how
to publish/subscribe the client does not have to approve or request from a central server to get
information. Unlike the traditional scenario (pull) where the client has to request every time it
wants to get information from the system. In general, push notifications are sent via Push
Notification Service (PNS) specific to each platform: such as Apple Push Notification Service
(APNs) for Apple, Google Cloud Message (GCM) for Android, and Microsoft Push Notification
Service (PNS) for Windows Phone. However, some of the algorithms for each civil servant are the
same (Laysha, 2015), namely, the three technologies use the Hypertext Transfer Protocol (HTTP)
protocol. HTTP is a standard protocol that functions to regulate the delivery of information in a
network. On the other hand, smartphone users must register their smartphone ID first with each
push technology before receiving information regularly. This is needed so that the sending of
notifications from the server to the client is not misdirected. The general notification delivery
mechanism for the three push technologies can be seen in Figure 1.
Figure 1. Push notification working mechanism
Figure 1 describes pushing notification technology using a push notification service (PNS) as
a broker on the web or android based smartphone. The working system of push notifications is as
follows:
1. Client registered with PNS as a broker,
2. Once registered, the broker provides an identification (token) to the client. Token
functions so that notifications are sent right to the token owner.
3. After the token is obtained from the broker, the client will send the token to the application
server.
4. The Token obtained from the client will be saved to the database.
5. The mechanism for sending notifications from the server to the client is:
6. Servers send a notification to the broker.
7. The broker receives notifications from the server and sent to the client according to the
token stored in the application server database.
ISRM 2021
157
WebSocket
WebSocket is an event-driven, full-duplex asynchronous communication channel for web
applications. WebSocket can provide real-time updates that previously used the long polling
method. The main advantage of using WebSocket is that it reduces resource requirements on both
the client and server-side. WebSocket uses HTTP as a transport mechanism, communication does
not end immediately after the response is received by the client, but as long as the connection is
still open the client and server can send messages to each other asynchronously (Estep, 2013).
Figure 2. WebSocket protocol working diagram
Figure 2 shows the handshake process between the client and the server on the WebSocket
protocol. First, the client will request a WebSocket connection on the server. Then the server will
respond to the connection request from the client. After the handshake process is formed, the
WebSocket protocol can send and receive data in a dual-channel model or different paths at the
same time (Ludovici & Calveras, 2015). The server will actively send data in the form of
notifications to the client. The process of sending notifications will continue to run until the
command is given to disconnect on the server-side. In this study, the WebSocket push server also
acts as a broker so that delivery is directly from the server to the client. When the client connection
with the server is lost, there is a notification on the server-side.
Website
The website can be interpreted as a collection of pages that can be used to display information
in the form of text, images, animation, sound, and combinations that are static or dynamic
depending on the value of data or parameters sent by the user to the webserver (Jader et al., 2016)
stored on the internet or called a webserver (Tasneem & Ammar, 2018). Research that has been
carried out is related to Sistem Terintegrasi Uji Kompetensi Online (SITUK) which applies web
push notification technology (Reyes et al. 2019), to conduct competency tests at the UPN "Veteran"
Jawa Timur LSP which results in web-based applications running well as expected (Wijaya, 2017).
Previous research that has been done also aims to make the system more efficient. Implementing
web notification technology, it is hoped that it can help asesor and asesi to get notifications when
a notification comes in. Web Push Notification is a method that sends data to a browser or mobile
device in real-time, which this web push notification can provide user convenience in the form of
information (Sulastri et al., 2016; Praba & Hariyanto, 2020). Information needed by users to use
the application, how the system works, such as short messages or short messages when messages
come in, a notification will appear to notify about data or messages that have been sent or when
ISRM 2021
158
the user's screen is run even if the user opens the web or not (Isikligil et al., 2017; Sulastri, et al.,
2016).
Web Push Notification is a method that sends data to a browser or mobile device in real-time,
which this web push notification can provide user convenience in the form of information
(Bidargaddi, et al. 2018; Sulastri et al., 2016). Information needed by users to use the application,
how the system works, such as short messages or short messages when messages come in, a
notification will appear to notify about data or messages that have been sent or when the user's
screen is run even if the user opens the web or not (Isikligil et al., 2017; Sulastri et al., 2016). Web
Push Notification is a method that sends data to a browser or mobile device in real-time, which
this web push notification can provide user convenience in the form of information (Reyes et al.,
2019; Paronen, 2015). Information needed by users to use the application, how the system works,
such as short messages or short messages when messages come in, a notification will appear to
notify about data or messages that have been sent or when the user's screen is run even if the user
opens the web or not (Isikligil et al., 2017; Sulastri, et al. 2016).
SITUK Application (Integrated System for Online Competency Test)
The competency test process begins with registration, both collectively and independently by
visiting the LSP of the University of Pembangunan Nasional "Veteran" Jawa Timur. This
registration will later provide information that will be submitted to the asesor or admin, who will
then be notified to the asesi if approved to take the competency test. This registration uses the
APL 1 form, the APL 1 form contains the data for the applicant or assessment and data on the
certification scheme to be followed along with the required evidence. After APL 1 is approved by
the admin or asesor, the asesi can continue to fill out the APL 2 form which contains data on the
independent assessment according to the selected scheme, which is to interpret themselves
according to the material to be tested. When working on APL 2, the asesi answered one of the
materials incompetently, then the asesi has been declared not to have passed the certification and
it is advisable to take part in training and debriefing before re-taking professional certification. If
the results of the competent assessment answers are all in APL 2, then the assessment is
considered eligible and can carry out a competency test (Masdiyasa, et al. 2020).
The implementation of the professional certification competency test consists of several
stages, namely written, verbal, practical, interviews, simulations, and others. If after the
competency test, and the asesi is declared competent, the asesi will receive a certification that is
recognized by the Badan Nasional Sertifikasi Profesi (BNSP). The process carried out starting from
registration to determining the results of the competency test still uses a manual system. A large
number of existing schemes also makes the process of grouping and distributing questions based
on schemes still not, organized and neatly stored. Asesi registration data is still manually inputted
by the admin which also makes this process take longer.
Material and Methods
This research was carried out at LSP UPN "Veteran" Jawa Timur, the determination of the
research location was based on considerations of data access, the data as the research had not
been processed much so the latest information was still scant. In this study, the method (Putri &
Wulandari, 2016) uses the waterfall model, which is a method with a linear sequence model or a
classical lifeline that approaches the plot sequentially. The stages of the method applied in this
research include analysis, design, coding, testing, and up to the support stage.
However, referring to the title of the research, this paper will discuss push notifications on
the SITUK application, where the device that acts as a server is the SITUK website at the domain
address and hosting: https://situk-web.igsindonesia.org/ with the operating system (Masdiyasa
et al. 2020). While the devices that act as clients are laptops or PCs and some smartphones only as
a comparison, of course with the Android operating system and laptops with the Windows
operating system or other operating systems on the asesi side. Implementation of notification
ISRM 2021
159
delivery mechanism (push service) using WebSocket on the server-side using a web-based
programming language. In addition, the implementation of push services on the server is
supported by using the Flask framework and the Socket.IO library. While on the client-side,
several devices have different needs. Client push notification implementation using WebSocket on
Android device using Flutter framework. While the implementation of the client on a laptop device
uses a web-based programming language as shown in Table 1. And Figure 3 is a block diagram of
the push notification test on the SITUK application.
Business process analysis Online Competency Test System (SITUK) Version 2
From the results of interviews conducted, several user requirements for the API (Masdiyasa,
et al. 2020) will be built, which are as follows:
1. Asesi candidates are required to register with the system before registering for the compe-
tency test.
2. The admin will determine the schedule for the competency test registration limit.
3. To register for the competency test scheme, asesi must register according to the schedule
provided by the admin on the system and must complete APL 1.
4. After the asesi registers for the competency certification test, the admin or asesor will verify
the APL 1 data and the attached requirements.
5. After the registration data has been verified, the admin will select an asesor for each session.
6. The next step is to fill out the APL 2 form, which is an independent assessment.
7. The asesor will check on APL 2, if there are results that are not yet competent then the asesi
is declared incompetent.
8. The asesor determines the date of the competency test on the participants being tested.
9. When the asesor conducts the competency test, the asesor does not need to write the results
of the verbal assessment, etc. to the system.
10. Asesor is required to fill in the form of observation checklist, portfolio evaluation checklist,
and competency assessment records.
11. The asesor decides whether the asesi is competent or not yet competent on the competency
assessment record form which the system will recheck the results of the competency test.
12. Admin can manage schema data, competency units, elements, and performance criteria on
the system.
13. The admin can manage the data for the competency test place.
Design and implementation push notification SITUK version 2
The simulation designed in this study is a simulation of the working mechanism of sending
notifications sent by the server to the client. In this study, the SSE protocol was used to be able to
compare the results between the two protocols according to the purpose of this study, but it would
be more directed to the use of WebSocket in the SITUK application (Masdiyasa et al. 2020), thus
this discussion is divided into two parts, namely the implementation of the WebSocket protocol
and the SSE protocol into the delivery mechanism on push notification on the same local network
(WLAN) as well as on the internet network, but more focused on the implementation of
WebSocket on the internet network through the domain situk-web.igsindonesia.org
As shown in Table 1, the device that acts as a server is the situk-web.igsindonesia.org domain.
The implementation of sending notifications using WebSocket on the server-side uses a web-
based programming language, namely javascript. It also uses the Flask framework and the
Socket.IO library. The Flask framework is used to design servers that are supported by the
Socket.IO library. For devices that act as clients are smartphones using the Android operating
system. The programming language used is javascript. Javascript serves to design web pages that
will be accessed by clients through a browser.
In the process of sending notifications on a per-client or broadcast basis, the user fills in the
register account column to be used as the value of the push notification variable and the link
ISRM 2021
160
variable. The data input variable in the notification process will have a value according to the
registration data. After the user enters the required notification data, the server will process the
data. The first process is to enter the value of the registration variable into the database. After the
process is successful, the next process is to send the notification to the connected client or user,
as shown in the flowchart of Figure 4, which is the order in which the notification is sent from the
server to the client.
Figure 4. The order of the process of sending notifications from the server to the client
In the SITUK-V2 business process, push notifications are carried out when asesi conducts the
registration process and is informed that asesi has passed the certification so that the push
notification process is carried out automatically by entering registration data and when the asesor
declares that the asesi has passed the competency test and a statement that the notification is sent
to the certain client to the server (Ma’rufi et al., 2020). Next, the server will add notification data
into the database. Then the notification will be sent to the client registered in the SITUK V2
database, as shown in Figure 5, which is an example of a notification made by the server to the
client (asesi) when registering (Sampurno et a., 2020).
Figure 5. Example of sending notifications made by server to client (asesi) when registering
Results and Discussion
Testing
Tests in this study include testing the maximum number of clients that can connect to the
server and testing the notification delivery time (delay) from the server to the client. The
notification delivery delay test is divided into 3 delivery conditions, including repeating
notification delivery, sending notifications with different packet sizes, and sending notifications
with different bandwidths. To be able to do the test, it takes a server and several clients to get
varied and more valid results. Table 2 is the hardware specifications used in this study as many as
ISRM 2021
161
362 clients, but only 102 clients who use PC or laptop computers and 15 clients who use mobile
devices can confirm.
Notification delivery repeat test based on package size and bandwidth
This test is carried out to prove that the WebSocket protocol can provide fast delivery services
when sending notifications in large quantities. All devices experienced consistent drops in each
scenario. However, the quality factor of the internet network and wireless devices in each device
affects sending a notification. Likewise, the Packet size test is carried out to determine the effect
of the delay in sending notifications from the server to all devices. And the notification delivery
test with a bandwidth of 1024 Kbps has a smaller average delay than the average notification
delivery test delay with a bandwidth of 512 Kbps.
Test result
This test is carried out by sending notifications to clients who carry out the registration
process and process graduation statements or statements of competence on 102 assessments
(clients) which are grouped into three (3) categories, namely asesi or asesor who use laptops or
PC, use Android mobiles and iOS mobiles. The results obtained from the test scenario will be
analyzed to get the average delay for each notification. The delay is obtained by using a timer
script and the average calculation operation on the client-side so there is no need to calculate the
average delay value for each scenario when processing data, as shown in Figure 6.
Table 2. Client hardware
Device Code
Device Type and Type
PC-Laptop
Mobile
L-PC-001
Laptop Asus A43E
L-PC-002
Laptop Asus FX-505DD
L-PC-003
Laptop Asus ROG GL553VD
L-PC-004
Laptop Dell Latitude
L-PC-005
Laptop Asus FX-505DT
L-PC-006
PC Ryzen 5 3600
L-PC-007
Laptop HP 1000
L-PC-008
Laptop Lenovo 330
L-PC-009
Laptop Asus X507UF
L-PC-010
Laptop MSI GL63 8RD
L-PC-011
Laptop Asus X441N
MB-A-001
Smartphone Lenovo A6000
MB-A-002
Xiaomi Redmi 5 Plus
Smartphone
MB-A-003
Vivo Y71 Smartphone
MB-A-004
Smartphone Oppo A7
MB-A-005
Asus Zenfone Max Pro M1
Smartphone
MB-I -001
iPhone XR
ISRM 2021
162
Figure 6. The result of sending notifications made by the server to the client (asesi) when registering and a
statement of competence
Figure 4 is a graph of the average delay in testing the 102 test clients, who registered and
declared competent on the SITUK-V2 application. Based on the graph in Figure 6, it can be seen
that there is a difference in delay between sending using WebSocket on a laptop or PC (on device
codes L-PC-001 to L-PC-011 and mobile devices (MB-A-001 to MB). -A-005 and MB-I-001), where
L-PC-xxx is the code for laptop and PCs, while MB-A-xxx and MB-I-xxx are the code for mobile
devices (Android and iOS). The delay between devices is 0.107s, while in terms of the package size
sent via notification during the registration process and the competency statement process, the
delay difference is 0.131s.
Conclusion
The implementation of the WebSocket protocol on push notifications in this study uses a
server https://situk-web.igsindonesia.org/, which also acts as a broker. The client is implemented
on two operating systems, namely Android, iOS, and Windows operating systems. The Android
operating system is implemented on smartphone devices by creating a web-based application and
Windows is implemented on laptop devices by creating a website page. On the server side, it can
be concluded that there are 102 clients connected to the server from the asesi and the asesor
session. Sending notifications in the registration process and competency statement process
causes delays in sending notifications based on the size of the package and the device used. This
is evidenced by the average delay in sending notifications on device usage of 0.107s. This is
because network traffic on smartphone devices is denser than network traffic on laptop devices.
While the average delay in sending notifications based on package size is 0.131s. The packet size
factor of a notification affects the delay in sending a notification.
Acknowledgment
Thank you to LPPM UPN “Veteran” Jawa Timur, for the research that has been given, so that
we can publish papers in several journals.
References
Bidargaddi, N., Pituch, T., Maaieh, H., Short, C., & Strecher, V. (2018). predicting which type of push notification content motivates users
to engage in a Self-Monitoring App. Preventive Medicine Reports, 11. Doi:10.1016/j.pmedr.2018.07.004.
Brüstel, J., & Preuss, T. (2012). A Universal push service for mobile devices. Sixth International Conference on Complex, Intelligent, and
Software Intensive Systems, 40-45. doi: 10.1109/CISIS.2012.105.
Estep, E. (2013). Mobile HTML5: Efficiency and performance of websockets and server-sent events. Sweden: Aalto University.
Hansen, J., Grønli, M., & Ghinea, G., (2012). Cloud to device push messaging on android: A case study. 26th International Conference on
Advanced Information Networking and Applications Workshops. pp. 1298-1303. doi: 10.1109/WAINA.2012.96.
ISRM 2021
163
Isikligil, E., Samakay, S., & Kılınç, D. (2017). A prototype framework for high performance push notifications. International Journal of
Computer Applications. 166, 8-11. 10.5120/ijca2017914122.
Jader, O., Zeebaree, S., & Zebari, R. (2019). A state of art survey for web server performance measurement and load balancing
mechanisms. International Journal of Scientific & Technology Research. 8. 535-543.
Laysha, A. (2015). Push-to-succeed: a look into Azure Notification Hubs for business.
Ludovici, A. & Calveras, A. (2015). A Proxy Design to Leverage the Interconnection of CoAP Wireless Sensor Networks with Web
Applications. Sensors (Basel, Switzerland), 15, 1217-44. 10.3390/s150101217.
Ma’rufi, M., Diyasa, I., & Sugiarto. (2020). Perancangan Sistem Uji Sertifikasi Kompetensi Berbasis GraphQL. Jurnal Informatika Dan
Sistem Informasi (JIFoSI), 1(2), 651-658.
Masdiyasa, I., Budiwitjaksono, G., M, H., Sampurno, I., & Mandenni, N. (2020). Graph-QL Responsibility Analysis at Integrated
Competency Certification Test System Base on Web Service. Lontar Komputer: Jurnal Ilmiah Teknologi Informasi, 11(2), 114-123.
doi:10.24843/LKJITI.2020.v11.i02.p05
Masdiyasa, I., Budiwitjaksono, G., M, H., & Sampurno, I. (2021). comparative analysis of rest and GraphQL technology on Nodejs-Based
Api Development. 5th International Seminar of Research Month 2020
Muhammad, P., Yahya, W., & Basuki, A. (2017). Analisis perbandingan kinerja protokol websocket dengan protokol SSE pada teknologi
push notification. Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, 2(6), 2235-2242.
Paronen, T. (2015). A web-based monitoring system for the industrial internet. Aalto University. School of Science.
Praba, A., & Hariyanto. (2020). Performansi web server Apache dan Nginx pada aplikasi penjualan online. Indonesian Journal on
Networking and Security, 9(3), 1-5. doi: http://dx.doi.org/10.2311/ijns.v9i3.1635
Putri, M., & Wulandari, D. (2016). Sistem Informasi Monitoring Siswa Berbasis Web Dan SMS Gateway pada SMK Negeri 37 Jakarta.
Jurnal Teknik Komputer AMIK BSI, II(2). 49–55.
Reyes, A., Martos, F., Rojas, R., Luque, R., & Molina, G. (2020). Effectiveness Of PUSH notifications from a mobile app for improving the
body composition of overweight or obese women. A Protocol of a Three-Armed Randomized Controlled Trial. BMC Medical
Informatics and Decision Making, 20. 10.1186/s12911-020-1058-7.
Sampurno, I., Susrama, I., & Sugiarto, (2020). Sistem terintegrasi uji kompetensi lembaga sertifikasi profesi. Jurnal Penelitian Politeknik
Penerbangan Surabaya Edisi XXIX, 5(3), 301-304.
Sulastri, H., Sevtiyuni, P., Rahmatulloh, A., & Sanjiwani, N. (2020). The Effect of Implementing Web Push Notification and Customer
Relationship Management in E-Commerce of Silver Craft MSME. 10.2991/aisr.k.200424.089.
Tasneem, S. & Ammar, R. (2012). Performance Study of a Distributed Web Server: An Analytical Approach. Journal of Software
Engineering and Applications. 05. 855-863. 10.4236/jsea.2012.511099.
Wijaya, H. (2017). Penerapan metode waterfall pada sistem informasi pendaftaran pasien rawat jalan berbasis Web Mobile. Jurnal
Sisfokom (Sistem Informasi Dan Komputer), 6(2), 80. https://doi.org/10.32736/sisfokom.v6i2.251
Yudianto, A., Pramukantoro, E., & Amron, K. (2017). Pengembangan push notification menggunakan websocket. Jurnal Pengembangan
Teknologi Informasi dan Ilmu Komputer, 1(1), 11-17.