We are going to add a custom button “Send Mail” to custom grid in magento2, on which chick we just show an form in popup which will be responsible to send an custom email to customer.

For this we will need to follow some simple steps-

1. Add customer_listing.xml file to your module in path –  app/code/Vendor/Module/view/adminhtml/ui_component/customer_listing.xml and add a custom button in this file

   <?xml version="1.0" encoding="UTF-8"?>
   <listing xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Ui:etc/ui_configuration.xsd">
       <columns name="customer_columns">
           <actionsColumn name="send_mail" class="Vendor\Module\Ui\Component\Listing\Columns\Sendmail">
               <argument name="data" xsi:type="array">
                   <item name="config" xsi:type="array">
                       <item name="component" xsi:type="string">Vendor_Module/js/grid/columns/sendmail</item>
                       <item name="indexField" xsi:type="string">entity_id</item>
                       <item name="sortable" xsi:type="boolean">false</item>
                       <item name="label" xsi:type="string" translate="true">Send Mail</item>
                       <item name="sortOrder" xsi:type="number">13</item>
                   </item>
               </argument>
           </actionsColumn>
       </columns>
   </listing>

2. As we want to show an form on click on this custom “Send Mail” button , now we need to create a ui component class Vendor\Module\Ui\Component\Listing\Columns\Sendmail –

   <?php
   namespace Vendor\Module\Ui\Component\Listing\Columns;
   
   use Magento\Framework\UrlInterface;
   use Magento\Framework\View\Element\UiComponent\ContextInterface;
   use Magento\Framework\View\Element\UiComponentFactory;
   use Magento\Ui\Component\Listing\Columns\Column;
   
   class Sendmail extends Column
   {
       /**
        * @var UrlInterface
        */
       protected $urlBuilder;
   
       /**
        * Constructor
        *
        * @param ContextInterface $context
        * @param UiComponentFactory $uiComponentFactory
        * @param UrlInterface $urlBuilder
        * @param array $components
        * @param array $data
        */
       public function __construct(
           ContextInterface $context,
           UiComponentFactory $uiComponentFactory,
           UrlInterface $urlBuilder,
           array $components = [],
           array $data = []
       ) {
           $this->urlBuilder = $urlBuilder;
           parent::__construct($context, $uiComponentFactory, $components, $data);
       }
   
       /**
        * Prepare Data Source
        *
        * @param array $dataSource
        * @return array
        */
       public function prepareDataSource(array $dataSource)
       {
           if (isset($dataSource['data']['items'])) {
               $fieldName = $this->getData('name');
               foreach ($dataSource['data']['items'] as & $item) { 
                   $item[$fieldName . '_html'] = "<button class='button'><span>Send Mail</span></button>";
                   $item[$fieldName . '_title'] = __('Please enter a message that you want to send to customer');
                   $item[$fieldName . '_submitlabel'] = __('Send');
                   $item[$fieldName . '_cancellabel'] = __('Reset');
                   $item[$fieldName . '_customerid'] = $item['entity_id'];
   
                   $item[$fieldName . '_formaction'] = $this->urlBuilder->getUrl('grid/customer/sendmail');
               }
           }
   
           return $dataSource;
       }
   }

3. Now we will need to create a js file sendmail.js in path app/code/Vendor/Module/view/base/web/js/grid/columns/ which we are using in this custom button.

   define([
       'Magento_Ui/js/grid/columns/column',
       'jquery',
       'mage/template',
       'text!Vendor_Module/templates/grid/cells/customer/sendmail.html',
       'Magento_Ui/js/modal/modal'
   ], function (Column, $, mageTemplate, sendmailPreviewTemplate) {
       'use strict';
   
       return Column.extend({
           defaults: {
               bodyTmpl: 'ui/grid/cells/html',
               fieldClass: {
                   'data-grid-html-cell': true
               }
           },
           gethtml: function (row) {
               return row[this.index + '_html'];
           },
           getFormaction: function (row) {
               return row[this.index + '_formaction'];
           },
           getCustomerid: function (row) {
               return row[this.index + '_customerid'];
           },
           getLabel: function (row) {
               return row[this.index + '_html']
           },
           getTitle: function (row) {
               return row[this.index + '_title']
           },
           getSubmitlabel: function (row) {
               return row[this.index + '_submitlabel']
           },
           getCancellabel: function (row) {
               return row[this.index + '_cancellabel']
           },
           preview: function (row) {
               var modalHtml = mageTemplate(
                   sendmailPreviewTemplate,
                   {
                       html: this.gethtml(row), 
                       title: this.getTitle(row), 
                       label: this.getLabel(row), 
                       formaction: this.getFormaction(row),
                       customerid: this.getCustomerid(row),
                       submitlabel: this.getSubmitlabel(row), 
                       cancellabel: this.getCancellabel(row), 
                       linkText: $.mage.__('Go to Details Page')
                   }
               );
               var previewPopup = $('<div/>').html(modalHtml);
               previewPopup.modal({
                   title: this.getTitle(row),
                   innerScroll: true,
                   modalClass: '_image-box',
                   buttons: []}).trigger('openModal');
           },
           getFieldHandler: function (row) {
               return this.preview.bind(this, row);
           }
       });
   });
   

4. Now we have to create a template file sendmail.html in path app/code/Vendor/Module/view/base/web/templates/grid/cells/customer/ which we are using in the above js file to create an send mail form

   <form id="send-mail-form-<%- customerid %>" method="get" enctype="multipart/form-data" action="<%- formaction %>">
       <div class="modal-body">
           <div class="bootbox-body">
               <textarea class="bootbox-input bootbox-input-text form-control required-entry" name="send_message" rows="4" cols="83"></textarea>
               <input type="hidden" name="entity_id" value="<%- customerid %>">         
           </div>
       </div>
       <div class="modal-footer">
           <span class="error"></span>
           <button type="reset" class="btn btn-default"><span><%- cancellabel %></span></button>
           <button type="submit" class="btn btn-primary"><span><%- submitlabel %></span></button>
           <span class="clear"></span>
       </div>
   </form>

on click on “Send Mail” button you will see a form in popup which will be responsible to send a custom mail to customer.

So in this way you can add any custom button or any custom column like action link, image to any grid in magento2.